1. Field of the Invention
The present invention relates generally to the field of optical microscopy and pertains more specifically to a fully automatic rapid microscope slide scanner.
2. Discussion of the Prior Art
One of the inherent limitations of optical microscopy is the tradeoff between the field of view, the portion of the sample that can be viewed through the eyepieces of a microscope, and the magnification at which the sample can be viewed. While higher magnification microscope objective lenses with higher numerical apertures (NA) provide the microscopist with an enlarged and often higher resolution image, the field of view decreases dramatically with increases in magnification, in proportion to the square of the magnification. Even at very low magnifications such as 1.25 times (1.25xc3x97), only a small area of a typical microscope slide can be viewed through the binoculars of a conventional microscope. The field of view limitation of optical microscopy requires that the microscopist manually scan a slide at low magnification to obtain an overall view of the sample or specimen. When an area of interest appears in one of the lower magnification fields of view, the microscopist manually selects a higher magnification objective lens to obtain an enlarged higher resolution view of a proportionately smaller area of the specimen. For samples such as histological specimens that are viewed by a pathologist, it is typical for the pathologist to frequently switch back and forth between a lower magnification objective lens with a larger field of view, for purposes of orienting himself or herself with respect to the specimen, and one or more higher magnification, smaller field of view objective lenses for purposes of viewing the sample in greater detail.
One approach to overcome the optical microscopy limitation of simultaneously achieving both a large field of view, and high magnification, is to capture multiple individual digital images from contiguous fields of view, thereby creating a large field of view image. A scanning system is used to move the sample, while a rectangular optical sensor such as an area scan charge-coupled device (CCD) camera captures an image of each field of view at the desired magnification. The process of assembling these smaller fields of view (hereinafter xe2x80x9cimage tilesxe2x80x9d) into one coherent image is called image tiling. Early image tiling systems, such as the system discussed in U.S. Pat. No. 4,760,385 (Jannson et al.) were based on creating a contiguous high resolution tiled image from approximately thirty-six individual video frame image tiles captured in a region of the sample that was previously and interactively selected by an operator. Similar but more sophisticated image tiling system have more recently become available. One such system is sold by Bacus Laboratories, Inc., Downers Grove, Ill., under the name Bacus Laboratories Inc., Slide Scanner (hereinafter xe2x80x9cBLISSxe2x80x9d). Elements of the BLISS system are described in Patent Cooperation Treaty publications WO 98/39728 and WO 98/44446.
The BLISS system is designed primarily for the anatomic pathologist who has a need to combine the anatomic orientation of a histological specimen that is obtained at very low magnification, together with several high magnification views of areas of the specimen that have been interactively selected by the pathologist from the low magnification tiled image, also referred to as a macro image. The BLISS system enables the pathologist to quickly flip back and forth between selected high resolution micro images of selected areas captured at 20xc3x97 or 40xc3x97, and a low resolution macro image captured at 1.25xc3x97, emulating in some sense the pathologist""s manual use of a conventional microscope. Alternatively, the BLISS system user interface provides separate split screens on a display monitor whereby the pathologist is shown an overall macro view and a marker showing where the current higher magnification view is located. A tiled image is constructed by assembling several adjacent, original microscope views at a first magnification to obtain an overall macro view of the specimen, together with several adjacent original microscope views at a higher magnification to create a combined data structure. The data structure is obtained by digitally scanning and storing the low magnification image tiles with their mapping coordinates and likewise, digitally scanning and storing higher magnification image tiles with their mapping coordinates. Furthermore, a pathologist may interactively select only those diagnostically significant areas of the specimen for digital scanning and storing to reduce significantly the number of image pixels stored at high resolution. The data structure, akin to a virtual microscope slide, may then be transferred to a remote viewer over a network such as the Internet. The remote user is thus provided with a series of abutted, tiled images, with each image tile being substantially equal to one small optical field of view at each of two different optical magnifications.
The BLISS system is integrated around a computer-controlled, automated microscope such as the Axioplan-2 microscope system sold by Carl Zeiss, Inc., Thornwood, N.Y. This type of high-end microscope has capabilities for computer-control of several subsystems, including the illumination subsystem, the focusing subsystem, the microscope objective lens subsystem, the filtering subsystem, as well as multiple field and condenser diaphragms or optical stops which may be used to achieve optimum Koehler illumination. Essentially, all moveable elements of the microscope can be controlled from the computer; and in principle, from a remote location via the Internet. Positions for all diaphragms and other settings such as focus and illumination level are stored by the computer, enabling microscope objective lenses to be changed without manual intervention. The BLISS system is also equipped with a computer controlled two-axis (x/y for left/right/up/down motion) translation stage that achieves 0.1 micrometer positioning accuracy using position encoders and closed-loop feedback control to provide superior positioning performance. A CCD camera with 752 by 480 pixels, and an image frame grabber are also integrated into the BLISS system.
Because it is based on image tiling, the BLISS system suffers from several known disadvantages of the image tiling approach. For example, a first disadvantage of the BLISS system is that it takes a long time, typically twenty minutes or longer to acquire the tiled data structures. These time estimates are without consideration for any additional delays that may be incurred during manual intervention, for example, prior to acquiring high magnification tiled images from selected areas of the low magnification macro image. Tiling involves moving a slide on a motorized stage, in discrete steps equal to the width of a single field of view, and with respect to a stationary area scan camera such as the CCD camera used by the BLISS system. An image tile is acquired at every step. Individual images are then tiled together to create a larger seamless image of the area of interest. Image tiling is relatively slow because of the need to minimize any significant relative motion between the sample and the camera while the image is captured. A major cause of relative motion is the settling time of the mechanical positioning stage after issuing sequential stop and go commands. To acquire images without unacceptable smearing requires waiting until the stage has settled, ideally to within less than one pixel. For example, at a 40xc3x97 magnification, the width of a single image tile captured by a one-half inch format CCD camera corresponds to 160 micrometers of the sample. At this magnification, each individual pixel in a 752-pixel wide CCD camera subtends approximately 0.2 micrometers of the sample. A single tiling step thus requires a relatively large 160 micrometer movement, with associated acceleration and deceleration of the mechanical stage. In order to avoid any smearing of the image, the image tile should be captured only after the mechanical stage has settled to less than one pixel, or about 0.2 micrometers, of motion. U.S. Pat. No. 5,912,699 (Hayenga et al.) addresses this well known settling time limitation of conventional image tiling systems by proposing an alternate method that combines image tiling using conventional area scan cameras with strobe light synchronization. The slow capture times of tiling systems, including the BLISS system, limits the practical utility of image tiling to a two-step process, with extensive manual intervention between the capture of an initial very low magnification macro image and the subsequent selection of small areas for higher magnification capture.
The slow acquisition time associated with tiling systems leads to a second disadvantage of the BLISS system, that being the need for manual intervention during the process of creating the tiled data structure. After pre-scanning a slide at a very low microscope objective lens magnification of 1.25xc3x97, the BLISS operator inspects the macro-image for relevant regions of interest to be scanned using a higher magnification objective lens. While one motivation for the manual intervention may be to limit the size of the final data structure, manual intervention is absolutely essential to define smaller areas which can be acquired in a reasonable time. For example, it would not be practical, because of acquisition time considerations, to use the BLISS system to scan an entire microscope slide at 20xc3x97 magnification. At a 20xc3x97 magnification, approximately 16,300 individual image tiles must be captured to digitize a two inch by one inch area of a microscope slide using a 752 by 480 pixel one-half inch format area scan CCD. Assuming further that it takes approximately one second to acquire each image tile, due in large part to the relatively long mechanical settling times associated with each of the 16,300 repeated stop-and-go commands, the total acquisition time would be four and one-half hours. At a 40xc3x97 magnification, the acquisition time would quadruple to eighteen hours. Even at a 10xc3x97 magnification the acquisition time would exceed one hour. However, at the BLISS system""s very low magnification of 1.25xc3x97, only 64 image tiles are needed to create a macro-image of a two inch by one inch area of a microscope slide. The total acquisition time for such a macroimage is about one minute.
Understanding now that the acquisition time limitations of any image tiling system require the capture of a very low magnification macro-image, followed by the interactive selection from this macro image of small areas to be captured at higher magnification, a third disadvantage of the BLISS system becomes apparent. This third disadvantage resides in the realization that locating areas of interest from a very low magnification macro-image is practically limited to samples in which anatomic reference information is available. The BLISS system thus has limited utility for non-histological samples such as Pap smears, because such cytological samples inherently lack any information about anatomic orientation. In such samples the cells are more or less randomly distributed over a large area of the microscope slide. Without the ability to define, using the macro image, the specific smaller regions of interest that are to be tiled at higher optical magnifications, the only alternative is to scan and digitize the entire sample. However, as described previously, the long acquisition times required by the image tiling method make this alternative virtually impractical. Stated differently, without manual intervention to define specific and significantly smaller areas of the microscope slide for image tiling at higher magnifications, an impossibility for cytological samples, a tiling approach has limited utility. One would prefer a system for scanning microscope slides which is fully automatic, without the need for manual intervention. Such a system would also be suitable for all types of microscope slides, regardless of whether or not the slide contains anatomic reference information.
A fourth disadvantage of the BLISS system is its complexity and expense. The BLISS system is based largely on off-the-shelf components, including a high-end, fully automated third-party microscope with multiple objective lenses and an expensive closed x/y stage control loop. The suggested end-user price of the BLISS system is well above $100,000. The multiple automated elements of the BLISS system represent a complicated system that, in spite of its extensive automation, may be difficult to operate and maintain. One would prefer a system for scanning microscope slides which is simple and reliable, and which can be made available for about one third of the cost of the BLISS system.
Inherent in the cost disadvantage of the BLISS system are several limitations of any microscope slide scanning system that is based on a conventional microscope. The most expensive component of the BLISS system is the automated microscope itself. One of the reasons for incorporating a fully automatic microscope into the BLISS system is the need for automatically changing many settings when the microscope objective lens turret is rotated automatically to change microscope objective lenses, for example, from 1.25xc3x97 to 40xc3x97. A typical microscope, upon changing the microscope objective lens, will have a different optimal plane of focus and require new settings for the field and condenser diaphragms to achieve Koehler illumination. Also, a different intensity of illumination will be needed to optimally fill the dynamic range of the CCD. The need for such extensive automation is eliminated if the requirement for changing microscope objective lenses can be eliminated. One would prefer a rapid scanning method that can not only overcome the field of view limitations of traditional optical microscopy but that can also eliminate the need for multiple microscope objective lenses, providing a substantial cost advantage over image tiling systems such as BLISS. The need for a single microscope objective lens is also closely related to eliminating the constraints imposed by the optics of a conventional microscope. One would prefer a system based on an optical design that ensures that microscope slides are scanned and digitized at diffraction-limited resolutions, that is, all possible spatial details available at the resolution of the microscope objective lens are captured by the digital image. Once a diffraction-limited digital image has been captured, degenerate lower resolution and magnification images can be created using standard computational algorithms.
In many microscopy applications it is necessary that the entire sample, or a large portion of a sample, be searched for defects or for the presence or absence of a special object or objects, for example, abnormal cells. Microscopy becomes very labor intensive when large portions of a sample, or even the entire sample, must be manually scanned at low resolutions, typically 10xc3x97 to 20xc3x97, in order to identify specific areas of interest for subsequent higher resolution interrogation. Extended manual screening or continued viewing of a single field causes eyestrain, fatigue, and visual habituation that can negatively affect productivity and accuracy. The problem of rapidly finding and locating relevant areas of interest for subsequent higher resolution interrogation has been addressed using conventional real-time scanning systems that combine microscopes with ancillary linear array detectors and automated positioning tables, all controlled by a computer. Some approaches, such as the system discussed in U.S. Pat. No. 5,922,282 (Ledley et al.), are based on storing the x/y stage coordinates of relevant objects found on regions of the physical slide to enable relocation of the object, in this case mycobacteria on a customized glass microscope slide. The x/y coordinates of the mycobacteria are obtained using specialized real-time pattern recognition circuitry that is applied to the intensity information measured by a line scan camera that is synchronized to a stage which is moved in relatively large five micrometer steps. Alternatively, an area scan sensor such as a video camera can be used as the basis for deriving the x/y coordinates of selected objects, in conjunction with similar circuitry. In this latter case, the stage is moved in larger steps corresponding to a complete image field, similar to the stage movements required by the tiling method. Focus is maintained using instantaneous automated focus control. An alternate system described in U.S. Pat. No. 4,700,298 (Palcic et al.) uses a linear array CCD attached to a commercially available microscope, with means for autofocus, to scan large areas for purposes of recording, in real time, the x/y coordinates of cells growing in a tissue flask. These known methods and systems are all based on the real-time analysis of digital information that is acquired and processed during the scanning process. In many cases, specialized circuitry is used to immediately process the intensity data that has been read out from the linear array detector, enabling a decision to be made in real-time. An alternative novel approach is to use a linear array sensor to rapidly assemble a large contiguous image of the entire microscope slide at optical resolutions sufficient for automating the labor intensive aspects of manual slide scanning. One would prefer a system that can be used, together with digital image processing methods, as an alternative to manual slide scanning.
Another common problem associated with manual scanning of microscope slides is that portions of a slide are easily missed during manual x/y scanning of a slide. Relocation to previously identified cells can be difficult, especially after the slide has been removed from the microscope. The problem of not missing any areas of a slide during manual x/y scanning has been addressed by position encoding quality assurance systems that record the x/y position and dwell time of areas of the physical slide that have been examined manually, thus highlighting areas of the slide that were missed or possibly viewed too quickly. U.S. Pat. No. 5,793,969 (Kamentsky et al.) discusses a method for quality assurance of a Pap smear slide that has been previously reviewed by a technologist. This method is based on recording the x/y stage coordinates of all fields visited by the technologist during the slide review, and creating an x/y map of relative slide dwell times.
A definite need exists for a simple and reliable system that can rapidly scan and digitize an entire microscope slide. The scanning and digitization should be performed at optical resolutions comparable to those used for the manual scanning of microscope slides, thereby enabling the application of image processing algorithms to the digital imagery data set, in lieu of, or in addition to, manually scanning an entire slide. Ideally, such a system should not require any type of manual intervention during the image acquisition process. Such a system should also be suitable for all types of microscope specimens, regardless of whether or not the slide contains anatomic reference information. Ideally, such a system should have a lower cost than conventional systems. Such a system should also not be constrained by the limitations and inherent cost of conventional off-the-shelf microscopes, enabling an optical design that allows the capture of diffraction-limited digital images. A primary purpose of the present invention is to solve these needs and to provide further, related advantages.
The present invention provides an apparatus for and a method of fully automatic rapid scanning and digitizing of an entire microscope sample, or a substantially large portion of a microscope sample, using a linear array detector synchronized with a positioning stage that is part of a computer controlled microscope slide scanner. The invention also provides a method for composing the image strips obtained from successive scans of the sample into a single contiguous digital image. The invention further provides a method for statically displaying sub-regions of this large digital image at different magnifications, together with a reduced magnification macro image of the entire sample. The invention also provides a method for dynamically displaying, with or without operator interaction, portions of the contiguous digital image. In one preferred embodiment of the invention, all elements of the scanner are part of a single-enclosure that has a primary connection to a network such as the Internet or a local intranet. In this embodiment, the preferred sample type is a microscope slide and the illumination and imaging optics are consistent with transmission mode optics optimized for diffraction-limited digital imaging.